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Studies have examined the impact of distraction on basic task performance (e.g., working 
memory, motor responses), yet research is lacking regarding its impact in the domain of 
think-aloud cognitive assessment, where the threat to assessment validity is high. The 
Articulated Thoughts in Simulated Situations think-aloud cognitive assessment paradigm 
was employed to address this issue. Participants listened to scenarios under three 
conditions (i.e., while answering trivia questions, playing a visual puzzle game, or with 
no experimental distractor). Their articulated thoughts were then content-analyzed both 
by the Linguistic Inquiry and Word Count (LIWC) program and by content analysis of 
emotion and cognitive processes conducted by trained coders. Distraction did not impact 
indices of emotion but did affect cognitive processes. Specifically, with the LIWC system, 
the trivia questions distraction condition resulted in significantly higher proportions of 
insight and causal words, and higher frequencies of non-fluencies (e.g., "uh" or "umm") 
and filler words (e.g., "like" or "you know"). Coder-rated content analysis found more 
disengagement and more misunderstanding particularly in the trivia questions distraction 
condition. A better understanding of how distraction disrupts the amount and type of 
cognitive engagement holds important implications for future studies employing cognitive 
assessment methods. 
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INTRODUCTION 

Given the limited capacity of human cognition, engaging in any 
non-automatic task (including any type of cognitive-affective 
assessment) requires that attention be selectively focused on 
relevant information while irrelevant information is ignored. 
Distractors are frequently thought of as external sensory stimuli 
(e.g., sights or sounds), but they may also be internal pro- 
cesses (e.g., one's mood states, cognitions, physical sensations). 
Distraction, also known as task disengagement, represents a 
potential threat to the reliability and validity of any assessment 
procedure by interfering with an individual's ability to complete 
a task at peak performance. Although the detrimental effects 
of distraction on a variety of more basic cognitive tasks (e.g., 
visuo-spatial and working memory tasks) are well known (e.g., 
Lavie, 2005; Tremblay et al., 2005), studies explicitly examining 
the effects of distraction on engagement in cognitive-affective 
think-aloud paradigms are notably absent. 

Cognitive-behavioral theory posits that the manner in 
which individuals interpret the world around them influences 
their behavioral and emotional responses. Acknowledging that 
thoughts are central to intra and interpersonal functioning as 
well as to psychopathology, psychologists have used a variety 
of questionnaires and assessment methods to capture cogni- 
tions and emotions. One such cognitive-emotional assessment 
approach is the Articulated Thoughts in Simulated Situations 
(ATSS) experimental paradigm. The term "paradigm" rather 



than "instrument" or "method" reflects that the ATSS is a basic 
approach to cognitive-emotional assessment and not a spe- 
cific assessment technique or instrument. The ATSS paradigm 
(Davison et al., 1983) was introduced as a way to access complex 
cognition and emotion in experimenter-controlled situations of 
considerable complexity. This think-aloud paradigm involves the 
research participant being asked to imagine that he/she is part of 
an audio-recorded situation and to verbalize his/her thoughts and 
emotional reactions periodically during pauses in the vignette. 
Investigators can tailor the paradigm's parameters to suit par- 
ticular experimental and clinical demands and preferences. As 
a result, the ATSS has been utilized in diverse areas of inquiry 
such as anger and aggression, interpersonal bias, psychotherapy 
process, eating disorders, alcohol use, maintenance of smoking 
cessation, depression, and social anxiety (for further detail, see 
reviews by Davison et al., 1997; Zanov and Davison, 2010). 

The assessment of emotions and thoughts is qualitatively dif- 
ferent from the assessment of more basic cognitive abilities. In 
particular, many cognitive ability tasks employ objective crite- 
ria such as response accuracy and speed to assess performance. 
However, subjective responses involving self-reported thoughts 
and feelings in the domain of cognitive-emotional assessment 
cannot be judged by the same performance-driven criteria. While 
it is necessary to exercise caution in drawing direct comparisons 
between the hypothesized effects of distraction in both cogni- 
tive domains, we also recognize the dearth of research into the 
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effects of distraction on cognitive-affective assessments. Thus, the 
current study draws from select research examining the effect 
of distraction on assessments of cognitive abilities to examine 
the effects of distraction on the think-aloud cognitive-affective 
assessment paradigm known as the ATSS. 

Research employing basic cognitive tasks reveals that, rather 
than being uniform across tasks, the impact of distraction varies 
based on the characteristics of the task and of the distractor. 
Specifically, the modality, perceptual and cognitive load of the 
task, as well as the modality and salience of the distractor have 
been found to be of importance (for a review, see Lavie, 2005). For 
example, a main determinant of the amount of interference a dis- 
tractor has on task performance is the level of cognitive load (or 
difficulty) of a task. According to Lavie et al. (2004), if a target task 
places a high load on working memory, fewer cognitive resources 
are left to maintain selective attention and thus mind-wandering 
in response to task-irrelevant stimuli becomes more likely. This 
load theory of cognitive control suggests that the more the dis- 
tractor competes with goal-directed behavior, the harder it is to 
maintain goal-relevant priorities (Lavie et al, 2004). Thus, if one 
is engaged in a complex cognitive task that is taxing to executive 
functioning, it becomes more difficult to sustain focus and ignore 
distractors. Indeed, studies employing paradigms that require 
switching between two tasks or performing two tasks simultane- 
ously (such as a manual bolt screwing task and random letter 
generation task performed simultaneously; e.g., Bourke, 1996; 
Monsell, 2003) document performance decrement in response 
accuracy and speed as compared to single task performance. 
Applied to think-aloud cognitive-affective assessment, this the- 
ory suggests that high-load distractors (e.g., multi-component 
tasks) may impact responding by slowing down performance 
(i.e., limiting the amount of articulated thoughts and feelings) 
and increasing confusion and misunderstanding of the presented 
scenarios. In turn, slowed performance and mental confusion 
regarding information being presented could negatively impact 
the quality of participants' responses and subsequent conclusions 
drawn from the data, without the experimenter knowing that 
such threats to validity have taken place. 

Studies that do not specifically examine task load suggest that 
shared input modality (e.g., visual or auditory) between simulta- 
neous tasks increases interference between the tasks. For example, 
simultaneous presentation of inputs from the same modality 
(visual or auditory) produces more task interference than the 
simultaneous presentation of inputs from different modalities 
(Duncan et al., 1997). In order to not overburden participants by 
making task performance extremely difficult, the current study 
uses visual presentation of distractors during the ATSS, the input 
for which is auditory. 

Semantic similarity of the information presented concurrently 
also increases interference (Hirst and Kalmar, 1987). For example, 
during a dichotomous listening task, participants demonstrated 
greater difficulty when they received the same type of input 
(letters or numbers) in both ears than when it was different 
(letters and numbers simultaneously). Previous research also sug- 
gests that auditory (spoken) and visual (written) presentation of 
semantic information activates some overlapping brain regions 
(e.g., Jobard et al., 2007; Buchweitz et al., 2009). That is, hearing 



and reading words activate similar neuronal pathways. This find- 
ing implies that a visual, semantic distractor is more likely to 
interfere with an open-ended think-aloud task than a visual, non- 
semantic distractor. The current study therefore includes both a 
semantic and non-semantic distractor. 

The aforementioned studies all utilized tasks that do not 
explicitly involve emotion-provoking stimuli. However, emotion 
has important consequences for attention. Emotional informa- 
tion (usually negatively-valenced) interferes with and frequently 
takes precedence over neutral information presented simulta- 
neously (e.g., Hartikainen et al., 2000; Ohman et al., 2001; 
Calvo et al., 2007). In situations that place constraints on atten- 
tional resources, attention is likely to be drawn toward emotional 
items, especially ones that may represent a threat (Mathews 
and Mackintosh, 1998), that have personal significance (Moray, 
1959; Mogg and Bradley, 1998), or that are congruent with one's 
mood state (Yovel and Mineka, 2005; Salters-Pedneault et al., 
2007). Furthermore, studies suggest that processing of emotional 
information may happen automatically, bypassing more effort- 
ful cognitive pathways (Zajonc, 1980). Consequently, processing 
self-relevant, emotion-laden information (such as a personally- 
relevant scenario eliciting negative emotion) may take precedence 
over other simultaneously occurring tasks. 

The current study seeks to examine the effects of task disen- 
gagement on responding to the ATSS paradigm by experimentally 
inducing distraction. To test the effects of distraction, partic- 
ipants were asked to verbalize their thoughts and feelings in 
response to three commonly used types of hypothetical scenar- 
ios in the ATSS literature (i.e., a neutral, an anxiety-provoking, 
and an anger-provoking scenario) in one of two experimentally 
manipulated distraction conditions: ( 1 ) while playing Tetris or (2) 
while answering trivia questions. A third condition involved no 
experimentally-induced distractor. The distractors were chosen to 
mimic two main types of distraction that are most likely to occur 
while a participant is engaging in a cognitive-emotional assess- 
ment task. Playing Tetris represents an exogenous, non-verbal 
distractor that calls for a motor response (i.e., similar to play- 
ing a non-verbal game on one's cell phone or interacting with 
an object in the environment). Answering trivia questions is a 
endogenous verbal distractor that places a load on working mem- 
ory and triggers thoughts relevant (or not) to the content of the 
trivia question (i.e., mimicking participants' engagement in a ver- 
bal task such as reading a text message, checking social media 
updates, etc.). 

Based on the reviewed literature, the following hypotheses 
were generated. First, given that negatively-valenced, self-relevant 
information captures attention and given extensive evidence from 
over four dozen ATSS experiments (see review by Zanov and 
Davison, 2010), we expected that individuals in all conditions 
would be able to engage in the task and produce emotionally- 
congruent verbalizations (i.e., anger in the anger-provoking sce- 
nario and anxiety/worry in the anxiety-provoking scenario). In 
other words, no differences were expected in terms of the degree 
of articulated negative emotions (namely, anger, and anxiety) in 
distracted vs. non-distracted individuals across all three ATSS 
scenarios. However, considering the high cognitive load of the 
ATSS paradigm and the limited processing capacity of human 
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cognition, we also expected the experimental distractions to detri- 
mentally impact certain types of performance. We hypothesized 
that this interference would manifest itself in higher levels of 
articulated distraction, disengagement, and misunderstanding of 
the scenarios. Furthermore, given the semantic nature of the 
ATSS, we also expected that the semantic distractor (i.e., reading 
trivia questions) would produce more interference with respect 
to these parameters than the non-semantic distractor (i.e., play- 
ing Tetris). Lastly, as studies have shown that people may be 
less affected by distractors if they are focusing on an emotional 
task, we hypothesized that individuals would display more artic- 
ulated distraction, disengagement, and misunderstanding of the 
scenarios in the neutral scenario than in the anger and anxiety 
scenarios. 

MATERIALS AND METHODS 
PARTICIPANTS 

Participants were 102 students recruited from the psychology sub- 
ject pool at the University of Southern California and received 
course credit for their participation in the study. Each partic- 
ipant attended a 1-h lab-based research session, during which 
they participated in the ATSS paradigm (see below; Davison 
et al., 1997). Prior to arrival, participants were randomized into 
three conditions: a Tetris distraction (where participants played 
a computerized Tetris game while listening to the audiotaped 
scenario), a Trivia distraction (where participants were required 
to read to themselves and then answer aloud as quickly as 
possible a trivia question with three multiple choice answers 
while listening to the audiotaped scenario), and a control con- 
dition (normal participation in the ATSS paradigm). Scenario 
order was counter-balanced across participants (for a total of 18 
randomizations: 3 distraction conditions X 6 counter-balanced 
scenario orders). Thirty-four participants (6 male) were ran- 
domized into the Tetris distraction, 28 participants (6 male) 
were randomized into the Trivia distraction, and 40 partici- 
pants (5 male) were randomized into the control condition. 
There were no significant differences in age, ethnicity, or edu- 
cation across conditions (all p > 0.05). The study protocol was 
approved by the local Human Subjects Institutional Review 
Board and all participants provided informed consent prior to 
participation. 

MEASURES AND PROCEDURE 

Procedure 

Instructions for participating in the ATSS paradigm were pre- 
sented both by audio and in written format simultaneously via 
desktop PC. To help improve the quality of responses elicited 
from the participant by the stimulus recordings, a practice sce- 
nario was used during which the experimenter provided feed- 
back about the participant's think-aloud performance (e.g., "Try 
to fill the whole thirty seconds of silence" and "Please discuss 
both your thoughts and your feelings as you imagine yourself 
in this scenario right now") to encourage maximal involvement 
in the procedure. After completing the ATSS paradigm, partic- 
ipants were debriefed by a research assistant and assessed for 
self-reported distraction and attentiveness over the course of the 
study. 



Articulated Thoughts in Simulated Situations 

Three audio scenarios were employed. Each scenario consisted of 
seven segments lasting approximately 20 s. After each individual 
segment, a tone signaled the beginning of a 30-s silence during 
which participants verbalized their thoughts and emotions as if 
they were in that scenario at that immediate moment in time. 
Distractions (as described above) were presented during the stim- 
ulus section but not during the 30-s period of silence during 
which participants articulated their thoughts and feelings. The 
stimulus recordings consisted of the following: 

Auto mechanic garage tape (anger-inducing scenario). The par- 
ticipant was asked to imagine that he/she had left work early to 
pick up his/her car at the mechanic. A female secretary greeted 
the participant and notified the manager that the participant had 
arrived for his/her car. During the course of the scenario as nar- 
rated throughout seven segments, the participant overheard the 
manager and mechanic discuss the status of the participant's car, 
which was not ready for pick up and in need of new parts. The 
manager suggested cheating the participant by charging more for 
parts as well as asking for more time to repair the car. 

Midterm tape (anxiety-inducing scenario). The participant was 
asked to imagine that it was the day of his/her psychology 
midterm and that he/she was sitting outside the room before the 
exam, overhearing two fellow students discuss the upcoming test. 
The two speakers discussed the extremely difficult nature of the 
course as well as last-minute changes to the test format that made 
the exam more challenging and intimidating. During the course 
of the tape, the participant found out that they had not received 
prior notice of exam changes and a review session through an 
electronic bulletin board. 

Cafeteria tape (neutral scenario). The participant was asked to 
imagine that he/she was waiting in line for lunch and overhearing 
two students in line discuss plans for the upcoming weekend. 

RESULTS 

TRAINED HUMAN CONTENT ANALYSIS 

Three undergraduate research assistants were trained on a total 
of five coding categories, including two emotional codes: (1) 
anger, hostility, and aggression; and (2) distress/anxiety/worry; 
and three distraction-specific codes: (3) overt distraction; (4) mis- 
understanding, and (5) disengagement. All coding categories were 
scored on a four-point (0-3) scale, considering both the amount 
and quality of expressions utilized by participants. Scores for 
each category were summed across segments for each scenario. 
Coding meetings were held weekly when coding initial partic- 
ipants to discuss coding challenges and inconsistencies. After 
this intensive training period, coders met approximately once a 
month to reduce coder drift. Coders showed very high absolute 
agreement for all codes: Anger intraclass correlation coefficient 
(ICC) = 0.978, Distress/ Anxiety ICC = 0.980, Overt Distraction 
ICC = 0.996, Misunderstanding ICC = 0.983, Disengagement 
ICC = 0.954. 

Anger, hostility, and aggression (AHA) captured statements 
and feelings of strong displeasure, antagonism, and/or "angry 
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annoyance." Distress/anxiety/worry referred to feelings of worry 
and nervousness. Overt distraction focused on direct statements 
of distraction or confusion by the participant. Misunderstanding 
gauged distraction based on accuracy of verbalizations to the sce- 
nario; this code required coders to attend to segment content 
and have a sense of what would and would not be appropri- 
ate/relevant (e.g., expressing anxiety about a psychology essay 
due, rather than a psychology midterm, which was the theme of 
one of the stimulus tapes). Disengagement was a judgment of how 
interested the participant was in the segment and how well they 
participated/worked with the material; this category reflects ver- 
balizations that were disjointed, unfocused, jumbled, bored, or 
reflected apathy. 

LINGUISTIC INQUIRY AND WORD COUNT CODING 

Transcripts for each participant were also coded through the 
Linguistic Inquiry and Word Count software program (LIWC; 
Pennebaker et al., 2007). LIWC has been used extensively to code 
written statements and verbal transcripts (for review, see Tausczik 
and Pennebaker, 2010). The LIWC program codes verbal mate- 
rial by identifying the percentage of words fitting into a particular 
code category. For the purposes of this study and given the nature 
of the scenarios presented, we used the LIWC emotional codes of 
anger (e.g., hate, kill, annoyed), and anxiety (e.g., worried, fearful, 
nervous) to measure emotional expression. 

To operationalize distraction through the LIWC program, 
we also examined cognitive process codes and spoken category 
codes. Cognitive process codes identify different facets of think- 
ing, including insight-oriented words (e.g., think, know, con- 
sider) and causation words (e.g., because, effect, hence), and may 
reflect processes of cognition that differ between experimentally 
distracted and non-distracted individuals. The spoken category 
codes used here were non-fluencies (e.g., errr, hmmm, umm) 
and fillers (e.g., blah, I mean, you know) and have been sug- 
gested to represent disorganized thinking process (Tausczik and 
Pennebaker, 20 10). We also considered the total number of words 
used by each participant as an indicator of distraction. 

Due to the non-normal nature of the main outcomes, non- 
parametric (i.e., Kruskal-Wallis) One- Way ANOVAs were con- 
ducted for each hypothesis, using distraction type or scenario 
type as the grouping variable. Non-parametric pairwise compar- 
isons were conducted for outcomes significantly differing across 
all groups in order to determine pairwise differences between 
individual groups with a two-tailed Bonferroni-corrected alpha 
level of 0.05. All analyses were conducted using SAS software (SAS 
version 9.2, SAS Institute, Cary, NC). 

MANIPULATION CHECK: DISTRACTION 

Distractor task scores were not analyzed due to the nature of 
the distractor tasks. The trivia questions were selected for their 
difficult and thought provoking nature; as a consequence, low 
performance is not indicative of low levels of distraction. The 
tetris distractor task was rate limited so that blocks fell at a 
constant pace. As a result, variability in score was also reduced 
and not necessarily indicative of engagement with the distractor. 
Consequently, in order to verify the impact of the experimentally- 
induced distractions on our think-aloud cognitive assessment 



paradigm, we examined both overt distraction (coded by trained 
research assistants) as well as self-reported distraction and atten- 
tiveness between the three conditions. Overt distraction differed 
significantly by distraction condition, Hp) = 7.19, p = 0.028. 
Pairwise comparison found that overt distraction was highest in 
the trivia condition, with significant differences between trivia 
and control, and marginal differences between trivia and tetris 
(and thus no statistically significant difference between the con- 
trol and tetris conditions). 

This pattern of results also was found in self-reported distrac- 
tion, with significant differences between conditions, F( 2 , 70) = 
8.12, p < 0.001 (self- reported distraction and attentiveness were 
normally distributed, hence the use of the F-statistic). Again, pair- 
wise comparisons found that distraction was highest in the trivia 
condition but no statistically significant difference between the 
control and tetris conditions was found. Self-reported amount of 
attention paid during the course of the study also significantly 
differed between groups, P(2, 69) = 12.27, p < 0.001. However, 
pairwise comparisons found that individuals in the control condi- 
tion were able to pay significantly more attention than in the tetris 
and trivia conditions. These analyses indicate that, as predicted, 
participants in the trivia condition felt greater distraction in their 
condition relative to participants in the other two conditions. 

MANIPULATION CHECK: EMOTIONAL CONGRUENCE 

The degree of anger and anxiety differed significantly by sce- 
nario types in the proportion of expressed anger, ffp) = 70.99, 
p < 0.001 and anxiety, H (2 ) = 82.71, p < 0.001. (Table 1 here) 
Table 1 displays the mean proportion of LIWC emotion codes and 
rated emotion codes by scenario type along with the overall and 
pairwise comparison analyses. Pairwise comparison revealed that 
the anger scenario elicited the most anger of all three scenarios, 
with no significant differences in anger between the neutral and 
anxiety scenarios. Similarly, the anxiety scenario elicited the most 
anxiety of all three scenarios; the anger and neutral scenarios did 
not differ significantly in proportion of anxiety words used. These 
results confirm that the emotional scenarios were successful in 
eliciting the desired emotional response from participants. 

WORD COUNT 

The number of words used across all scenarios significantly dif- 
fered across conditions, H( 2 ) = 6.36, p = 0.042; mean words 
(SD) used by condition per scenario were 495.70 (118.66) for 
the control condition, 466.28 (109.49) for the trivia condition, 
and 466.26 (119.06) for the tetris condition. Pairwise compar- 
isons indicated that participants in the control condition used 
significantly more words than participants in the two distraction 
conditions, which did not significantly differ from one another. 
Consequently, we used proportion of words in each LIWC cate- 
gory as our main outcome rather than number of words, given 
the significant differences in word count between conditions. 

IMPACT OF DISTRACTION ON VERBALIZED EMOTION 

Using data from the LIWC program, no significant differences 
were found among the three conditions of distraction (tetris, 
trivia, no distraction) for verbalized anxiety, H(2) = 2.53, p = 
0.282, or anger, H(2) = 0.39, p = 0.824 across all scenarios. 
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Table 1 | Mean (Standard Deviation) LIWC emotion code scores 8 and total coder-rated emotion scores for the anger (ANG), anxiety (ANX), and 
neutral (NEU) ATSS scenarios. 



Emotion code Anger scenario Anxiety scenario Neutral scenario Scenario main effect Non-parametric 

(ANG) (ANX) (NEU) (Kruskal-Wallis pairwise 

non-parametric ANOVA) comparisons 



LIWC anger 


1.36 (1.18) 


0.49 (0.67) 


0.40 (0.48) 


H i2) 


= 70.99, p < 0.001 


ANG > ANX, NEU 


LIWC anxiety 


0.49 (0.43) 


1.57 (1.52) 


0.39 (0.46) 


H i2) 


= 82.71, p < 0.001 


ANX > ANG, NEU 


Rated anger 


8.36 (4.01) 


1.21 (1.53) 


2.21 (2.58) 


H i2) 


= 70.99, p < 0.001 


ANG > ANX, NEU 


Rated anxiety 


1.31 (1.59) 


6.98 (4.60) 


1.40 (2.04) 


H i2) 


= 82.71, p < 0.001 


ANX > ANG, NEU 



"LIWC scores are in percentage of words out of the entire participant's transcript. 



Similarly, no significant differences were found among the three 
conditions for coder-rated anger, Hm = 3.46, p = 0.178 or anx- 
iety, H(2) = 0.013, p = 0.994. These results suggest that level of 
distraction did not affect the emotional involvement of partici- 
pants in the three scenarios. 

IMPACT OF DISTRACTION ON COGNITIVE ENGAGEMENT 

(Table 2 here) To examine cognitive engagement in the scenarios, 
analyses were conducted to test if there were differences between 
condition on LIWC codes for insight and causal words, as well 
as the LIWC codes of fillers and non-fluencies. Condition was 
significantly associated with proportion of insight words used, 
H(2) = 6.97, p = 0.031, and with proportion of causal words 
used,H(2) = 15.12,p < 0.001. Table 2 displays the mean propor- 
tion of cognitive process and spoken category codes by condition 
along with the overall and pairwise comparison analyses. Pairwise 
comparisons showed that participants in the trivia questions dis- 
traction produced a significantly higher proportion of insight and 
causal words than participants in the tetris and non-distraction 
conditions, but that there was no difference between the latter 
two conditions in these two codes. These results suggest that the 
trivia condition significantly impacted cognitive engagement in 
the scenarios. 

Examination of the spoken category codes indicated that con- 
dition was significantly associated with use of non-fluencies, 
H( 2 ) = 17.17, p < 0.001 and with fillers, H( 2 ) = 8.99, p = 0.011. 
Pairwise comparisons again showed that individuals in the 
trivia distraction condition verbalized a higher proportion of 
non-fluencies than did participants in both the tetris and non- 
distraction conditions and that there were no significant differ- 
ences between the tetris and non-distraction conditions. Pairwise 
comparisons for the impact of distraction condition on filler 
words used indicated that the control condition elicited signifi- 
cantly more fillers than the tetris condition, with no difference 
between the trivia and control conditions. These analyses imply 
that the trivia condition may result in more disorganized thought 
than the tetris or non-distraction conditions. 

Distraction condition was also associated with significant dif- 
ferences in disengagement, ffp) = 7.08, p = 0.029 and in mis- 
understanding, H(2) = 15.20, p < 0.001. Pairwise comparisons 
found that individuals in the trivia distraction exhibited more 
disengagement than participants in the control condition, with 
no significant difference between the tetris and control con- 
dition. Individuals in the trivia distraction condition showed 



significantly higher misunderstanding than both the tetris and 
control condition participants. These results indicate that partici- 
pants in the trivia condition display less engagement in the think- 
aloud assessment than participants in the other two conditions. 

IMPACT OF SCENARIO TYPE ON COGNITIVE ENGAGEMENT 

Scenario type was also related to proportion of insight words uti- 
lized, H( 2 ) = 51.62, p < 0.001, as well as causal words, H( 2 ) = 
18.33, p < 0.001 but not non-fluencies, H m = 5.17, p = 0.078 
or filler words, H(2) = 4.93, p = 0.085. Pairwise comparisons 
indicate that the anxiety scenario provoked significantly greater 
use of insight words than the anger and neutral scenarios. The 
neutral scenario also was associated with greater use of causal 
words compared to the anger and anxiety scenarios. These results 
suggest that the different scenarios may elicit different patterns of 
cognitive engagement. 

Scenario type was also significantly associated with coder- 
rated misunderstanding, Hp) = 16.71, p < 0.001 but not disen- 
gagement, Hi 2) = 1-38, p = 0.502. Pairwise comparisons found 
that the anger scenario was associated with significantly greater 
misunderstanding compared to the anxiety and neutral scenar- 
ios, with no significant difference between the anxiety and neutral 
scenarios. This result may indicate that understanding the anger 
scenario is more difficult than understanding the anxiety and 
neutral scenarios. 

DISCUSSION 

While the impact of distraction on task performance has been 
well studied, its impact on cognitive assessment is less well- 
documented, particularly regarding think-aloud assessment, and 
emotion-provoking stimuli. This empirical neglect is surprising 
given that task disengagement has often been raised by crit- 
ics as a threat to assessment validity, though primarily directed 
toward think-aloud methods. The current study examined this 
issue by experimentally manipulating distraction for individuals 
participating in a think-aloud assessment method known as the 
Articulated Thoughts in Simulated Situations paradigm. 

The first aim was to assess the impact of distraction on ver- 
balized emotions in response to three scenarios — one designed 
to induce anger, one to induce anxiety, and one to be relatively 
emotion-neutral. The scenarios were found to elicit the intended 
emotions among all individuals (e.g., the anger scenario being 
associated with the largest proportion of anger expressions and 
coder-rated anger), and distraction did not affect the proportion 
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Table 2 | Mean (Standard Deviation) LIWC cognitive processing and spoken category code Scores 3 and total coder-rated disengagement and 
misunderstanding for the trivia question (TRV), tetris (TET), and control distraction (CTL) conditions. 

Code Trivia question Tetris distraction Control condition Distraction main effect Non-parametric 

distraction (TRV) (TET) (CTL) (Kruskal-Wallis pairwise 

non-parametric ANOVA) comparisons 



Insight words 


3.83 (2.21) 


3.20 (1.49) 


3.15 (1.48) 


H [2) 


= 6.97, p = 0.031 


TRV 


> 


TET, CTL 




Causal words 


1.91 (0.91) 


1.53 (0.92) 


1.50 (0.81) 


H i2) 


= 15.12, p < 0.001 


TRV 


> 


TET CTL 




Non-fluencies 


2.95 (2.15) 


1.79 (1.51) 


2.02 (1.46) 


H (2) 


= 17.17, p < 0.001 


TRV 


> 


TET CTL 




Fillers 


1.67 (1.40) 


1.34 (1.48) 


1.73 (1.54) 


H [2) 


= 8.99, p = 0.011 


CTL 


> 


TET CTL = 


TRV 


Disengagement 


7.04 (4.85) 


5.93 (4.18) 


5.17 (3.70) 


H [2) 


= 7.08, p = 0.029 


TRV 


> 


TET TET = 


CTL 


Misunderstanding 


1.69 (2.27) 


0.94 (2.14) 


0.60 (1.22) 


H {2) 


= 15.20, p < 0.001 


TRV 


> 


TET CTL 




Mean words per scenario 


466.28 (109.49) 


466.26 (119.06) 


495.70 (118.66) 


H{2) 


= 6.36, p = 0.042 


CTL 


> 


TET TRV 





LIWC scores are in percentage of words out of the entire participant's transcript. 



of emotions expressed. That is, think-aloud cognitive assessment 
of emotion in this study was not significantly impacted by task 
disengagement, regardless of distractor modality. This finding 
supports the notion that emotionally salient stimuli are robust 
in the context of distraction and take attentional precedence over 
neutral stimuli (Ohman et al, 2001; Calvo et al, 2007). Our 
findings also support research by Zajonc (1980) that affective 
judgments and reactions have an innate quality and are not nec- 
essarily contingent upon extensive cognitive processing (or even 
recognition of the stimulus). Affective reactions may thus be less 
disrupted by distractors that interfere with cognitive encoding. In 
other words, individuals may experience a fairly automatic emo- 
tional response to a stimulus without having extensively encoded 
the characteristics of the stimulus (as might be the case in the 
presence of a distractor). 

This phenomenon is well-documented in studies of fear reac- 
tions that occur automatically and do not depend on higher- 
level cortical pathways (for an overview, see LeDoux, 2003; 
Vuilleumier, 2005) as well as in the domain of impression for- 
mation, where individuals have been found to quickly and auto- 
matically form judgments about a person after very brief exposure 
to his/her face without effortful encoding of his/her characteris- 
tics (e.g., Willis and Todorov, 2006; Todorov et al., 2009). As a 
result, individuals may have an emotional reaction, especially in 
response to threatening stimuli, without having to accurately per- 
ceive or consciously think about the stimuli. Thus, despite being 
under conditions of distraction, participants in this study may 
have been able to react with anger and anxiety to the presented 
scenarios, as accurate perception and sophisticated cognitive pro- 
cessing of all of the details of the situation were not necessary in 
order to have an affective response. 

Furthermore, affective information is readily conveyed via 
non-verbal vocal cues, such as the tone of a speaker's voice (e.g., 
Argyle et al, 1970). Given that all three of the experimental 
scenarios used in this study included overheard conversations 
between two speakers, affective information is likely to have been 
easily gleaned by participants from non-verbal vocal cues even 
when the processing of the verbal content of the overheard con- 
versations was disrupted. Indeed, participants are capable of iden- 
tifying the emotional valence of speech even when its content is 
digitally masked (e.g., Scherer et al., 1972). Thus, our participants 



are likely to have gleaned and reacted to emotional information 
from vocal cues even in the presence of distraction. 

Results of the current study indicate that emotion-focused 
cognitive assessment, at least in the case of the think-aloud 
paradigm used in this study, appears not to be significantly 
impacted by wandering attention during stimulus presentation 
immediately prior to assessment. However, emotional distractors 
have been found to negatively impact performance on cogni- 
tive (e.g., working memory) tasks (Dolcos and McCarthy, 2006; 
Dolcos et al., 2008; Denkova et al., 2010), due in part to dis- 
ruption in brain regions involved in attentional processes (e.g., 
the dorsolateral prefrontal cortex; Dolcos and McCarthy, 2006). 
Affectively-based distractions might have a stronger impact on 
cognitive and/or affective assessments, particularly given the 
strength of emotional distractors relative to non-emotional 
distractors (Dolcos et al., 2008). Thus, the extent to which 
emotion-focused cognitive assessment is disrupted by emotional 
or valenced distractions deserves further investigation and needs 
to include the use of different cognitive tasks, as well as various 
cognitive and/or affective paradigms. 

The second aim of this study was to examine the impact of 
task disengagement on cognitive processes by considering differ- 
ences in the use of causal words, insight-oriented words, filler 
words, and non-fluencies, as well as coder-rated disengagement 
and misunderstanding. The hypotheses were partially supported. 
Distraction appeared to significantly affect the organization of 
participants' thought, though this effect was confined primarily 
to the trivia distraction condition. Unexpectedly, distracted indi- 
viduals used more of these types of mental organization words. 
Tausczik and Pennebaker (2010) put forth that causal and insight 
words "can suggest the active process of reappraisal" (p. 36); in 
this study, causal and insight words may have been utilized to re- 
process some of the information the participant was able to hear 
in order to form a more coherent and cohesive narrative based on 
limited information. The notion of narrative formation through 
greater employment of causal words is supported by Boals and 
Klein (2005), who posited the use of causal words as indicative of 
an individual processing thoughts about an event and attempting 
to create "causal connections" (p. 263). As distracted participants 
in the ATSS paradigm might not have access to complete por- 
tions of the audio stimuli, they may be forced to process chunks 
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of the scenario during their think-aloud response, organizing 
information as it is recalled from short-term memory. 

The trivia questions distraction condition also resulted in 
greater proportions of non-fluencies than the other conditions. 
Use of non-fluencies like "umm" and "uhh" has been associated 
with increased cognitive load (Smith and Clark, 1993; Sporer and 
Schwandt, 2006) and has been found to occur more when fabri- 
cating information (i.e., lying; Vrij et al, 2011). These behaviors 
are in line with reorienting from distraction in an assessment sit- 
uation; some individuals may try to process the limited informa- 
tion when not distracted in the context of the overall assessment 
(e.g., scenario stimuli), while other individuals might attempt to 
fabricate plausible responses to the assessment prompts based on 
their understanding of the content to that point in the assessment. 

In both of these possible situations during think-aloud assess- 
ment, a large amount of cognitive resources is required to simul- 
taneously process information or fabricate responses while also 
responding aloud to the prompt with their thoughts and feel- 
ings. Based on this premise, greater cognitive resources ought to 
be associated with resiliency in the face of distraction. Meys and 
Sanderson (2013) found that individuals with better cognitive 
resources (in the form of greater working memory capacity) were 
more resistant to distraction during a running arithmetic task. 
Working memory may represent an individual difference factor 
that buffers against distraction across assessment paradigms and 
should be examined in future studies on the impact of distrac- 
tion on task performance. In addition, it is possible that the ATSS 
procedure is relatively immune to these distractors. 

While the trivia distraction led to significant differences on 
these primary outcomes, the Tetris distraction manipulation gen- 
erally did not differ significantly from the control condition on 
these outcomes. As the trivia distractor was a verbally based 
manipulation, the overlap in the linguistic domain between the 
distraction and the stimuli may have resulted in competing pro- 
cesses and greater interference. Indeed, in a study of memory 
for visually presented digits with audio presentation of irrele- 
vant digit, word, and non-word distractors, Salame and Baddeley 
(1982) showed that recall accuracy declined the most in the 
presence of digits and phonologically similar word distractors. 

Taken together, the findings of the current study support 
the idea that while distractors like phone-based video games 
might not significantly impact performance or engagement in 
self-report assessments, other types of distractors like texting 
may harm the fidelity of responses acquired from respondents, 
regardless of assessment modality (e.g., questionnaire, interview, 
think-aloud, etc.). This area of research is largely unexplored. 
Previous research on distraction has almost exclusively studied 
the impact of distraction on tasks whose outcomes are typi- 
cally based on some element of time. Only two studies to our 
knowledge have looked at distraction on assessments that used 
outcomes that were more qualitative or less time-dependent in 
nature. Dixon and Salley (2006) utilized a fast mapping (i.e., 
word-learning) protocol whereby children were presented five 
objects, four that were known exemplars and one that was a novel 
object. After given some time to explore the objects, children were 
asked to identify objects when provided the name. Novel labels 
(e.g., "dax" or "noop") were used for the novel exemplar and 



repeated in order to improve the learning process. A second tray 
was presented thereafter, with four known objects and two novel 
objects, one taxonomically related and one completely unrelated 
object (i.e., a foil). Two distraction conditions were utilized: one 
condition that involved increasing cognitive load via the addi- 
tion of an additional familiar object during the learning phase, 
and another that involved sudden distractions in the environment 
(e.g., a stranger walking into the room and reading a book, or a 
mechanical toy being operated in view of the child). Both dis- 
traction conditions impaired novel word acquisition. In another 
study, Thrift (2012) used an information visualization task to 
examine the impact of environmental distraction on the ability to 
accurately discern patterns and trends in data presented to partic- 
ipants. Subjecting participants to a noisy environment resulted in 
decreased task performance accuracy. Both of these studies sup- 
port the current findings that distraction can significantly impact 
performance in the context of assessment. 

LIMITATIONS 

The current study utilized a sample composed of college students. 
It is unclear the extent to which these results generalize to other 
age groups and educational backgrounds, particularly given that 
cognitive functioning (e.g., processing speed and working mem- 
ory) and thus the impact of distraction may change over the 
lifespan. 

Most studies examining the impact of distraction on task per- 
formance use distractors that have correct responses, allowing 
for response accuracy to be used as a measure of distraction. 
The present research did not have a similar check in place due 
to the nature of the distraction conditions (e.g., trivia ques- 
tions were selected for their difficulty and thought-provoking 
nature, meaning that most individuals did not select the right 
answer). However, the distraction manipulations were designed 
to be "heavy-handed" enough to test the upper limits, as it were, 
of the effects of distraction in a think-aloud assessment method. 
They were simultaneously designed to not overly burdensome 
on the participant, to avoid floor effects in performance. Results 
suggest that the modality of distraction may be critical in deter- 
mining whether a distractor is inconsequential or impairing for 
individuals engaged in an assessment. 

Lastly, the non-parametric nature of the data, in general, 
and consequent use of non-parametric analyses like the Kruskal- 
Wallis One-Way ANOVA, specifically, precludes the reporting of 
effect size coefficients in these analyses. In addition, due to the 
non-normality of the data, the study is likely underpowered to 
adequately detect interaction effects. Further experimental stud- 
ies with larger sample sizes are suggested to test for interaction 
effects with appropriate power. 

STRENGTHS. CONCLUSIONS, AND FUTURE DIRECTIONS 

The current study is among the first to examine the impact of 
task disengagement on cognitive assessment. Studies of distrac- 
tion have overwhelmingly focused on the impact of distraction 
on task performance. In addition, distraction was experimentally 
manipulated in this study, allowing for causal inferences based 
on our findings. Moreover, we used two methods of content- 
analysis for the responses provided to this think-aloud cognitive 
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assessment paradigm, with a consistent pattern of results across 
both content-analyses, which strengthened the reliability of the 
results. 

On the whole, one has little to no idea what the social and 
environmental context is when a participant responds to an 
online assessment. Less attended-to aspects of the testing environ- 
ment (such as ambient temperature or type of background noise 
present) have been found to have significant impacts on engage- 
ment and performance (e.g., Hancock, 1984; Enander, 1987; 
Hancock and Warm, 1989) and are thus important to consider in 
assessment, where individuals seek to gain accurate information 
regarding an individual's current state in a particular domain of 
functioning. 

We also found that the distraction conditions did not dif- 
fer in terms of the proportion of emotional responses elicited, 
indicating that distraction did not significantly impact the level 
of emotional intensity/responsivity to the scenarios. This is an 
important finding, as it suggests that even in the context of 
distraction, intended emotional responses can be elicited and 
assessed in a valid manner. For future studies employing cogni- 
tive assessment methods, more systematic consideration of how 
distraction can and does disrupt cognitive engagement is neces- 
sary, whether in the form of additional measures of engagement 
or use of attention tracking devices such as eye tracking programs. 

The present research indicates a need for more empirical atten- 
tion to be directed toward understanding the impact of distrac- 
tion on cognitive assessment given that these kinds of assessments 
are often administered without regard to participant environment 
or engagement. Distraction can negatively impact both timed 
and non-timed cognitive tasks (e.g., word-learning tasks; Dixon 
and Salley, 2006), suggesting that assessment, in all its forms, 
ought to attend more to the impact of environmental distractions. 
Ignoring or disregarding the possible impact of distractors on 
assessment validity becomes particularly concerning as more and 
more research involves online administration of measures, instru- 
ments, and tasks. Study participants maybe engaged in a number 
of different extraneous tasks that may result in drawing attention 
away from the desired engagement in responding. As described 
above, these distractions may negatively impact the amount and 
type of cognitive engagement that takes place, thereby raising 
questions about validity in the various assessment approaches. 
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